1. Technical Field
Present invention embodiments relate to information integration, and more specifically, to avoiding restart of an Extract, Transform, and Load (ETL) process in response to an error.
2. Discussion of the Related Art
Many organizations have adopted distributed data warehouse systems that can store data on a large number of machines and accommodate large amounts of information. Every day, millions of Extract, Transform, and Load (ETL) jobs send data to these data warehouses.
An ETL job may include multiple stages and processes running on many machines. Typically, an ETL job uses a combination of pipeline and partition parallelism, in which data is partitioned and moved through different pipelines in order to improve performance.
Some ETL jobs fail and need to be restarted or resubmitted. For example, the target database within a warehouse may fail to accept a set of records in the middle of a job because a capacity limit is reached or a hardware failure occurs. This may cause the entire job to abort. The user may have to resubmit the job after correcting the failure in order for the data to be successfully processed and submitted to the destination database.